Skip to content

Make TimedeltaIndex +/- pd.NaT return TimedeltaIndex #19139

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jan 16, 2018

Conversation

jbrockmendel
Copy link
Member

@jbrockmendel
Copy link
Member Author

Travis http error

@@ -302,14 +302,14 @@ def test_nat_arithmetic_index():
tm.assert_index_equal(left - right, exp)
tm.assert_index_equal(right - left, exp)

# timedelta
# timedelta # GH#19124
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add similar testing for Series here as well

Copy link
Member Author

@jbrockmendel jbrockmendel Jan 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shoot this surfaces a new bug; I'll xfail for now.

@jreback jreback added Datetime Datetime data dtype Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Timedelta Timedelta data type labels Jan 9, 2018
@jreback jreback added this to the 0.23.0 milestone Jan 9, 2018
tm.assert_index_equal(NaT - tdi, tdi_nat)


@pytest.mark.xfail(reason='NaT - Series returns NaT. This behavior was '
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you just parametrize around tdi and Series, too much repeated code (you can xfail that case using pytest.param)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls do this here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are pinging but not doing what I ask.

@jreback
Copy link
Contributor

jreback commented Jan 10, 2018

you need to rebase

@jbrockmendel
Copy link
Member Author

Looks like Travis is timing out in a bunch of places. Will hold off on pushing for now. Once this gets in I've got a patch ready for #19158 that will un-xfail that one test.

@codecov
Copy link

codecov bot commented Jan 10, 2018

Codecov Report

Merging #19139 into master will decrease coverage by <.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #19139      +/-   ##
==========================================
- Coverage   91.52%   91.52%   -0.01%     
==========================================
  Files         147      147              
  Lines       48827    48795      -32     
==========================================
- Hits        44691    44661      -30     
+ Misses       4136     4134       -2
Flag Coverage Δ
#multiple 89.9% <100%> (-0.01%) ⬇️
#single 41.61% <0%> (-0.09%) ⬇️
Impacted Files Coverage Δ
pandas/core/indexes/timedeltas.py 90.6% <100%> (-0.04%) ⬇️
pandas/core/dtypes/common.py 94.58% <0%> (-0.74%) ⬇️
pandas/core/indexes/interval.py 92.29% <0%> (-0.09%) ⬇️
pandas/util/testing.py 84.41% <0%> (-0.04%) ⬇️
pandas/core/dtypes/dtypes.py 96.11% <0%> (-0.02%) ⬇️
pandas/core/series.py 94.61% <0%> (ø) ⬆️
pandas/core/dtypes/cast.py 88.35% <0%> (+0.36%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aa9e002...5a54a92. Read the comment docs.

@jbrockmendel
Copy link
Member Author

ping. Once this goes in I can fix the bug that surfaced in pd.NaT - Series

tm.assert_index_equal(NaT - tdi, tdi_nat)


@pytest.mark.xfail(reason='NaT - Series returns NaT. This behavior was '
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls do this here.

@jbrockmendel
Copy link
Member Author

can you just parametrize around tdi and Series, too much repeated code (you can xfail that case using pytest.param)

Just tried this and pytest is raising errors at collect-time, complaining about pytest.param. This would be a non-issue if weren't for the xfail. After this branch is merged I've got a branch ready that fixes the xfailing error. Since that PR will have to change this test anyway, let's parametrize it in that PR.

@jbrockmendel
Copy link
Member Author

ping

tm.assert_index_equal(NaT - tdi, tdi_nat)


@pytest.mark.xfail(reason='NaT - Series returns NaT. This behavior was '
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are pinging but not doing what I ask.

@jbrockmendel
Copy link
Member Author

Please read my previous comment.

@jreback
Copy link
Contributor

jreback commented Jan 13, 2018

I read your comment. its doesn't make any sense.

@jbrockmendel
Copy link
Member Author

I read your comment. its doesn't make any sense.

Then ask for clarification, buddy. Just repeating the request without acknowledging that I tried to implement and discuss it is pretty discouraging.

Implementing the parameterization you're asking for looks like this:

@pytest.mark.parametrize('box, assert_func', [
    (TimedeltaIndex, tm.assert_index_equal),
    pytest.param((Series, tm.assert_series_equal),
                 marks=pytest.mark.xfail(reason='NaT - Series returns NaT'))
])
def test_nat_arithmetic_td64_vector(box, assert_func):
    [...]

(Even if it didn't cause an error, this is 5 lines of boilerplate to eliminate a 6 line test.)

python -m pytest pandas/tests/scalar/test_nat.py then raises at test-collection time:

/usr/local/lib/python2.7/site-packages/_pytest/python.py:832: in parametrize
    param.values, argnames))
E   ValueError: In "parametrize" the number of values (((<class 'pandas.core.series.Series'>, <function assert_series_equal at 0x10c0a6140>),)) must be equal to the number of names (['box', 'assert_func'])

So the options here are:

  • Wait for this to get fixed on pytest's end
  • kludge some other boilerplate to avoid needing to use pytest.param here
  • Merge as is, at which point I can make a PR that fixes the bug that makes the xfail necessary, and that PR can parametrize the test.

@jreback
Copy link
Contributor

jreback commented Jan 13, 2018

@pytest.mark.parametrize('box, assert_func', [
    (TimedeltaIndex, tm.assert_index_equal),
    pytest.param(Series, tm.assert_series_equal,
                 marks=pytest.mark.xfail(reason='NaT - Series returns NaT'))
])
def test_nat_arithmetic_td64_vector(box, assert_func):
    [...]

pass as args not a tuple. I agree its a little bit odd that pytest doesn't accept a tuple here, and I know that this is adding an xfail that you are going to remove later, but would rather do that than miss adding the tests with coverage.

@jbrockmendel
Copy link
Member Author

pass as args not a tuple.

Not sure what you mean here.

this is adding an xfail that you are going to remove later, but would rather do that than miss adding the tests with coverage.

The xfail and the coverage exist in the PR-as-is. If you can clarify the "args not a tuple" thing I'll try to implement it, since that is a faster way of getting bugs fixed than arguing about it. But compromising with me on this is an even faster way to get even more bugs fixed, and it ends with the same parametrized tests you have in mind.

@jreback
Copy link
Contributor

jreback commented Jan 13, 2018

do this

pytest.param(Series, tm.assert_series_equal,
                 marks=pytest.mark.xfail(reason='NaT - Series returns NaT'))

not this

pytest.param((Series, tm.assert_series_equal),
                 marks=pytest.mark.xfail(reason='NaT - Series returns NaT'))

@jbrockmendel jbrockmendel mentioned this pull request Jan 15, 2018
4 tasks
@jreback jreback merged commit 5e3cd25 into pandas-dev:master Jan 16, 2018
@jreback
Copy link
Contributor

jreback commented Jan 16, 2018

thank you for fixing!

jbrockmendel added a commit to jbrockmendel/pandas that referenced this pull request Jan 16, 2018
@jbrockmendel jbrockmendel deleted the tdi_pm_nat branch January 23, 2018 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Datetime Datetime data dtype Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Timedelta Timedelta data type
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TimedeltaIndex +/- NaT inconsistency with Series[timedelta64]+/- NaT
2 participants